{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Untitled3.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/arbi11/CompEM1/blob/master/NLP.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w_sFkzqk61g6",
        "colab_type": "code",
        "outputId": "6c8df6e9-e2b6-4fae-c34b-327b8cc78c7f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eusP0iNy68wA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sample_data = [\"Today is a cold Sunday morning. I am at the Nashville School of Law. \\\n",
        "               I am here for PyTennessee where I can learn more about Python.\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8SYiwPg3iUe",
        "colab_type": "text"
      },
      "source": [
        "## Sentence segmentation\n",
        "Also called sentence tokenization or sentence boundary disambiguation, breaks up sentences by deciding where a sentence starts and ends. Challenges include recognizing ambiguous puncutation marks. Use ```sent_tokenize``` from ```nltk.tokenize```. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JWYojqSJ7AM_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.tokenize import sent_tokenize\n",
        "\n",
        "def get_sent_tokens(data):\n",
        "    \"\"\"Sentence tokenization\"\"\"\n",
        "    sentences = []\n",
        "    for sent in data:\n",
        "        sentences.extend(sent_tokenize(sent))\n",
        "    print(sentences)\n",
        "    return sentences\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n4L1JVir7DAX",
        "colab_type": "code",
        "outputId": "d396b370-73f3-4d2e-ef2e-12d58dbd9591",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "sample_sentences = get_sent_tokens(sample_data)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['Today is a cold Sunday morning.', 'I am at the Nashville School of Law.', 'I am here for PyTennessee where I can learn more about Python.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_JBPYSXk3iUk",
        "colab_type": "text"
      },
      "source": [
        "## Word tokenization\n",
        "Similar to sentence tokenization, but works on words. Use ```word_tokenize``` from ```nltk.tokenize```. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lnf-oJ-07pt3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "def get_word_tokens(sentences):\n",
        "    '''Word tokenization'''\n",
        "    words = []\n",
        "    for sent in sentences:\n",
        "        words.extend(word_tokenize(sent))\n",
        "    print(words)\n",
        "    return(words)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FAkRoreu8CNv",
        "colab_type": "code",
        "outputId": "a476ce35-930c-4ab1-94b2-5208bbe454dd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "sample_words = get_word_tokens(sample_sentences)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['Today', 'is', 'a', 'cold', 'Sunday', 'morning', '.', 'I', 'am', 'at', 'the', 'Nashville', 'School', 'of', 'Law', '.', 'I', 'am', 'here', 'for', 'PyTennessee', 'where', 'I', 'can', 'learn', 'more', 'about', 'Python', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G0vX_ntP3iUo",
        "colab_type": "text"
      },
      "source": [
        "## Frequency distribution\n",
        "Calculates the frequency distribution for each word in the data. Use ```nltk.probability``` from ```FreqDist``` and ```matplotlib```."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xh6ITlKp8FWP",
        "colab_type": "code",
        "outputId": "d89977c2-e1f3-49c0-e78d-269e5b123f57",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 510
        }
      },
      "source": [
        "import matplotlib\n",
        "from nltk.probability import FreqDist\n",
        "matplotlib.use('TkAgg') \n",
        "\n",
        "def plot_freq_dist(words, num_words = 20):\n",
        "    fdist = FreqDist(words)\n",
        "    fdist.plot(num_words,cumulative=False)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "error",
          "ename": "ImportError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-13-e934415723ff>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprobability\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mFreqDist\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'TkAgg'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mplot_freq_dist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_words\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m20\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/matplotlib/__init__.py\u001b[0m in \u001b[0;36muse\u001b[0;34m(arg, warn, force)\u001b[0m\n\u001b[1;32m   1389\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mforce\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1390\u001b[0m             \u001b[0;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpyplot\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mswitch_backend\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1391\u001b[0;31m             \u001b[0mswitch_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1392\u001b[0m     \u001b[0;31m# Finally if pyplot is not imported update both rcParams and\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1393\u001b[0m     \u001b[0;31m# rcDefaults so restoring the defaults later with rcdefaults\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/matplotlib/pyplot.py\u001b[0m in \u001b[0;36mswitch_backend\u001b[0;34m(newbackend)\u001b[0m\n\u001b[1;32m    220\u001b[0m                 \u001b[0;34m\"Cannot load backend {!r} which requires the {!r} interactive \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    221\u001b[0m                 \"framework, as {!r} is currently running\".format(\n\u001b[0;32m--> 222\u001b[0;31m                     newbackend, required_framework, current_framework))\n\u001b[0m\u001b[1;32m    223\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    224\u001b[0m     \u001b[0mrcParams\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'backend'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrcParamsDefault\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'backend'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnewbackend\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mImportError\u001b[0m: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'headless' is currently running",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g3LNZE-98iNg",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3IrqJ593iUu",
        "colab_type": "text"
      },
      "source": [
        "## Cleaning the data\n",
        "Real world data is often messy. You can do a bunch of preprocessing to ensure the data is clean, like:\n",
        "- Removing special characters\n",
        "- Removing stopwords\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uqEbfKyL8NVX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "def remove_special_characters(sentences, remove_digits=False):\n",
        "    clean_sentences = []\n",
        "    for sent in sentences:\n",
        "        pattern = r'/[^\\w-]|_/' if not remove_digits else r'[^a-zA-Z\\s]'  # Regex needs correction\n",
        "        clean_text = re.sub(pattern, '', sent)\n",
        "        clean_sentences.append(clean_text)\n",
        "    print(clean_sentences)\n",
        "    return clean_sentences"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y9TWfCn88lJW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "7ee130e8-31c3-4fff-926a-9c29acc09b65"
      },
      "source": [
        "remove_special_characters(sample_sentences)\n"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['Today is a cold Sunday morning.', 'I am at the Nashville School of Law.', 'I am here for PyTennessee where I can learn more about Python.']\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Today is a cold Sunday morning.',\n",
              " 'I am at the Nashville School of Law.',\n",
              " 'I am here for PyTennessee where I can learn more about Python.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gm90zCnj8qDf",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-x2AO1Ol3iU0",
        "colab_type": "text"
      },
      "source": [
        "## Text processing\n",
        "Text processing approaches like stemming and lemmatization help in reducing inflectional forms of words. \n",
        "### Stemming\n",
        "Stemming tries to cut off at the ends of the words in the hope of deriving the base form. Use ```PorterStemmer``` from ```ntlk.stem```."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FCW4F-bN8nCX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Stemming and lemmatization\n",
        "from nltk.stem import PorterStemmer\n",
        "\n",
        "def get_stems(words):\n",
        "    ps = PorterStemmer()\n",
        "    stems = []\n",
        "    for word in words:\n",
        "        stems.append(ps.stem(word))\n",
        "    print(stems)\n",
        "    return stems"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gTCg303_8tan",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f4242661-a9f9-4ae9-807d-2832e4d4d878"
      },
      "source": [
        "sample_stems = get_stems(sample_words)\n"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['today', 'is', 'a', 'cold', 'sunday', 'morn', '.', 'I', 'am', 'at', 'the', 'nashvil', 'school', 'of', 'law', '.', 'I', 'am', 'here', 'for', 'pytennesse', 'where', 'I', 'can', 'learn', 'more', 'about', 'python', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uzI034F18xXI",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-aeUliXM3iU8",
        "colab_type": "text"
      },
      "source": [
        "### Lemmatization [ISSUE]\n",
        "Lemmatization groups different inflected forms of a words so they can be mapped to the same base. \n",
        "More complex than stemming, context of words is also analyzed. Uses WordNet which is a lexical English database. \n",
        "Use ```WordNetLemmatizer``` from ```nltk.stem```.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "65vBa0m28ufX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e65d9210-0db7-4c97-d404-636ed4ef03df"
      },
      "source": [
        "nltk.download('wordnet')\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "def get_lemma(words):\n",
        "    wordnet_lemmatizer = WordNetLemmatizer()\n",
        "    lemma = []\n",
        "    for word in words:\n",
        "        lemma.append(wordnet_lemmatizer.lemmatize(\"becoming\")) # Warning: Lemmatizer needs a POS tag or else it treats it as a noun and doesn't change it\n",
        "    print(lemma)\n",
        "    return(lemma)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DCp8C3dZ80rv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "d485d560-d0d3-426e-85c5-3b67fa7ef2ea"
      },
      "source": [
        "sample_lemma = get_lemma(sample_words)\n"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming', 'becoming']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5-irx1FO830X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "bf715265-4e18-4111-ad97-412720c9dc2a"
      },
      "source": [
        "print(sample_lemma)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cEnBwEid87gv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PuaOwZaR3iVC",
        "colab_type": "text"
      },
      "source": [
        "## POS tagging\n",
        "The English language is formed of different parts of speech (POS) like nouns, verbs, pronouns, adjectives, etc. POS tagging analyzes the words in a sentences and associates it with a POS tag depending on the way it is used. Also called grammatical tagging or word-category disambiguation. Use ```nltk.pos_tag```. There are different types of tagsets used with the most common being the Penn Treebank tagset and the Universal tagset. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "io469Edw3iVJ",
        "colab_type": "text"
      },
      "source": [
        "![Image of Yaktocat](https://i.stack.imgur.com/FhcKV.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MHQcMzEB9Xdg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "138c2152-2e90-4384-f436-561afe2c6fb0"
      },
      "source": [
        "nltk.download('averaged_perceptron_tagger')\n",
        "\n",
        "def get_pos_tags(words):\n",
        "    tags=[]\n",
        "    for word in words:\n",
        "        tags.append(nltk.pos_tag([word]))\n",
        "    print(tags)\n",
        "    return tags"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xAjZqqqG9YOn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "70a6674f-12f8-48fa-e149-8d91eb34113d"
      },
      "source": [
        "sample_tags = get_pos_tags(sample_words)\n"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[('Today', 'NN')], [('is', 'VBZ')], [('a', 'DT')], [('cold', 'NN')], [('Sunday', 'NNP')], [('morning', 'NN')], [('.', '.')], [('I', 'PRP')], [('am', 'VBP')], [('at', 'IN')], [('the', 'DT')], [('Nashville', 'NNP')], [('School', 'NN')], [('of', 'IN')], [('Law', 'NN')], [('.', '.')], [('I', 'PRP')], [('am', 'VBP')], [('here', 'RB')], [('for', 'IN')], [('PyTennessee', 'NN')], [('where', 'WRB')], [('I', 'PRP')], [('can', 'MD')], [('learn', 'NN')], [('more', 'RBR')], [('about', 'IN')], [('Python', 'NN')], [('.', '.')]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vf27WQWq9akP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZxgwL4rM6BnQ",
        "colab_type": "text"
      },
      "source": [
        "## Named entity recognition\n",
        "Use NER to identify entities like person, organization, city, etc. Helpful in redacting PII. More details here: https://www.nltk.org/book/ch07.html\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "54Re9U039sgO",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9rHoufw23iVL",
        "colab_type": "text"
      },
      "source": [
        "## Bag of words\n",
        "Bag of words is an approach for text feature extraction. Just imagine a bag of popcorn, \n",
        "and each popcorn kernel represents a word that is present in the text. Each sentence can be represented as a vector\n",
        "of all the words present in a vocabulary. If a word is present in the sentence, it is 1, otherwise 0.\n",
        "\n",
        "![Image of Yaktocat](https://cdn-images-1.medium.com/max/1600/1*zMdHVQQ7HYv_mMZ5Ne-2yQ.png)\n",
        "\n",
        "## TF-IDF\n",
        "Term-frequency inverse document frequency assigns scores to words inside a document. Commonly occuring words in all documents would have less weightage.\n",
        "![Image of Yaktocat](http://www.bloter.net/wp-content/uploads/2016/09/td-idf-graphic.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w3bv9E749v-H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sklearn\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "def get_bag_of_words(sentences):\n",
        "    vectorizer = CountVectorizer()\n",
        "    print(vectorizer.fit_transform(sentences).todense())\n",
        "    print(vectorizer.vocabulary_) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D1uvy-Z09yi2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "d1595c70-122b-42cc-9386-a0ac8f8684c4"
      },
      "source": [
        "get_bag_of_words(sample_data)\n"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]]\n",
            "{'today': 19, 'is': 7, 'cold': 4, 'sunday': 17, 'morning': 11, 'am': 1, 'at': 2, 'the': 18, 'nashville': 12, 'school': 16, 'of': 13, 'law': 8, 'here': 6, 'for': 5, 'pytennessee': 14, 'where': 20, 'can': 3, 'learn': 9, 'more': 10, 'about': 0, 'python': 15}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6HYoeYuTAO7w",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X1WMtI4W3iVb",
        "colab_type": "text"
      },
      "source": [
        "## Sentiment analysis\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nOPdcUXNAOh4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import nltk\n",
        "nltk.download('vader_lexicon')\n",
        "from nltk.sentiment.vader import SentimentIntensityAnalyzer \n",
        "  \n",
        "restaurant_reviews = [\"Great place to visit in Nashville.\",\n",
        "\"The food took too long to come, the service was slow.\",\n",
        "\"Everything was amazing.\",\n",
        "\"Place closed down a month ago.\",\n",
        "\"Had to wait in line for an hour, but the food was worth the wait.\",\n",
        "]\n",
        "  \n",
        "sentiment_analyzer = SentimentIntensityAnalyzer()\n",
        "for sentence in restaurant_reviews:\n",
        "     print(sentence)\n",
        "     sentiment_score = sentiment_analyzer.polarity_scores(sentence)\n",
        "     for score in sentiment_score:\n",
        "         print('{0}: {1},' .format(score, sentiment_score[score]), end='')\n",
        "     print()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U2-14oWNAXUg",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tCYt4BJS3iVb",
        "colab_type": "text"
      },
      "source": [
        "## Word embeddings - Word2Vec\n",
        "Vector space model - represent words and sentences as vectors to get semantic relationships. \n",
        "\n",
        "![Image of Yaktocat](http://www.flyml.net/wp-content/uploads/2016/11/w2v-3-samples.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "omtT4Zl2AVyw",
        "colab_type": "text"
      },
      "source": [
        "We demonstrate the following functions:\n",
        "\n",
        "\n",
        "1.   Train the word embeddings using brown corpus;\n",
        "2.   Load the pre-trained model and perform simple tasks; and\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VmDI0iWY9zQe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gensim\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iGrHLKlX-ig3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "59af260c-a905-419e-d0e8-d0fdd60e94a8"
      },
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "nltk.download('brown')\n",
        "from nltk.corpus import brown\n",
        "model = gensim.models.Word2Vec(brown.sents())"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package brown to /root/nltk_data...\n",
            "[nltk_data]   Package brown is already up-to-date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EDBg68qW-kQP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.save('brown.embedding')\n",
        "new_model = gensim.models.Word2Vec.load('brown.embedding')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JsxKii-O-1pn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5ea17d69-c877-4d52-c350-ddceea7addf2"
      },
      "source": [
        "len(new_model['university'])\n"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "100"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8g2d1fhf-5Hw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5f1d6d12-b4d7-494f-f83e-a8661ac72618"
      },
      "source": [
        "new_model.similarity('university','school') > 0.3\n"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RsaTGcBo-8g3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "8c8f5925-7ae0-4e3e-c658-23c8815f647b"
      },
      "source": [
        "nltk.download('word2vec_sample')\n",
        "from nltk.data import find\n",
        "word2vec_sample = str(find('models/word2vec_sample/pruned.word2vec.txt'))\n",
        "model = gensim.models.KeyedVectors.load_word2vec_format(word2vec_sample, binary=False)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package word2vec_sample to /root/nltk_data...\n",
            "[nltk_data]   Unzipping models/word2vec_sample.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xA6fBeuK_i-X",
        "colab_type": "text"
      },
      "source": [
        "We pruned the model to only include the most common words (~44k words).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NCTv283m_KKo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b1131f57-8cfa-4297-83c5-1838b53f4b90"
      },
      "source": [
        "len(model.vocab)\n",
        "\n"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "43981"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LkH35y6U_hPn",
        "colab_type": "text"
      },
      "source": [
        "Each word is represented in the space of 300 dimensions:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CrIB0Lbj_ZZo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "26b58d2a-2cd7-4440-8b3e-ee63990cca15"
      },
      "source": [
        "len(model['university'])\n"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "300"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FIwEzQeW_fk4",
        "colab_type": "text"
      },
      "source": [
        "Finding the top n words that are similar to a target word is simple. The result is the list of n words with the score.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JHamNE-S_cHf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "04deb12e-0f2e-4cb3-ca0f-bfbb7683ea94"
      },
      "source": [
        "model.most_similar(positive=['university'], topn = 3)\n"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('universities', 0.7003918886184692),\n",
              " ('faculty', 0.6780906915664673),\n",
              " ('undergraduate', 0.6587096452713013)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hmeRVfF6_eNX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "fe89fb14-8f0c-4373-cef3-bbede5565fdd"
      },
      "source": [
        "model.doesnt_match('breakfast cereal dinner lunch'.split())\n"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'cereal'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M1DNcRV0_oqA",
        "colab_type": "text"
      },
      "source": [
        "Mikolov et al. (2013) figured out that word embedding captures much of syntactic and semantic regularities. For example, the vector 'King - Man + Woman' is close to 'Queen' and 'Germany - Berlin + Paris' is close to 'France'."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m-fHL3aS_mjA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c1d3c7a7-85e1-46ed-c5a9-71ea37a0187e"
      },
      "source": [
        "model.most_similar(positive=['woman','king'], negative=['man'], topn = 1)\n"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('queen', 0.7118192911148071)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EiZHKORs_qlX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "43182a79-9697-4cca-8d2d-60f4d8070764"
      },
      "source": [
        "model.most_similar(positive=['Paris','Germany'], negative=['Berlin'], topn = 1)\n"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('France', 0.7884092330932617)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yU8XhBXG_sU_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}